Nowadays, users/customers of telecommunications networks and international regulatory bodies demand an extremely high quality of service with little or no periods of service failure or down time. Accordingly, many attempts have been made by equipment manufactures to develop the design of switching nodes that produce hitless and graceful restart when control plane software upgrade occurs in the telecommunication networks, especially in optical telecommunications networks. Graceful restart is only applicable to new generation switching nodes where the separation of data and control planes are implemented.
The RSVP (resource reservation protocol) graceful restart allows a router (a switching node) undergoing a restart to inform its adjacent neighbours of its condition. The restarting router requests a-grace period from the neighbour or peer, which can then cooperate with the restarting router. The restarting router can still forward MPLS (multi-protocol label switching) traffic during the restart period. The restart is not visible to the rest of the network. For the restarting router, the RSVP graceful restart maintains the path installed by RSVP and the allocated labels, so that traffic continues to be forwarded without disruption. This is done quickly enough to reduce or eliminate the impact on neighbouring nodes. For the router's neighbours, the neighbouring routers must have the RSVP graceful restart helper mode enabled, thus allowing those to assist a router attempting to restart RSVP.
All RSVP graceful restart procedures are timer-based for both restart and recovery. During the recovery time, a restarting node attempts to recover its lost states with assistance from its neighbours. The neighbour of the restarting node needs to send the PATH messages with the recovery labels to the restarting node within a period of one half of the recovery time. The restarting node considers its graceful restart complete after its advertised recovery time. Currently there is no way for RSVP to determine that it has completed a restart procedure, other than after a fixed timeout.
FIG. 1 of the prior art shows a telecommunications network 100 having a number of nodes, Node A 105 to Node I 145, and links, 107, 113, 117, 123, 124, 126, 129, 133, 136, 137, 139, 141, 142, 143, 144, 147, and 153, connecting the nodes. If the data forwarding between nodes is operating normally and a node control plane restarts, due to crash, software upgrade, or the control channel between a pair of nodes restarts, then all LSPs (label switched paths) traversing the node are terminated. This causes a major traffic disruption inside the network. With graceful restart, the control plane can be recovered without disrupting the data plane, that is, no disruption to data/user traffic. As shown in FIG. 1, two LSPs are going through Node C 115. One LSP is going through the links 107 from Node A 105 to Node B 110, 113 from Node B 110 to Node C 115, 117 from Node C 115 to Node D 120, and 123 from Node D 120 to Node E 125. Another LSP is going through the links 137 from Node G 135 to Node H 140, 143 from Node H 140 to Node C 115, 147 from Node C 115 to Node D 120, and 153 from Node D 120 to Node F 130. Node B 110, Node D 120, and Node H 140 have knowledge about the labels that are used for data forwarding on Node C 115. Node C 115 advertises the graceful restart capability to the neighbouring Node B 110, Node H 140, and Node D 120. If the control plane on Node C 115 has crashed and if the data forwarding is operating normally, Node B 110, Node H 140, and Node D 120 would not be impacted and will keep the LSPs running through 139, 141, 142, and 144 links intact. After detecting that Node C 115 is up again, Node B 110, Node D 120, and Node H 140 send label information to Node C 115 to help its recovery.
Prior art teaches different approaches of achieving graceful restart for traditional multi-protocol label switching (TMPLS) including state copy method, protocol method with network management system (NMS) assistance, protocol method with minimal traffic hit, and protocol method with zero traffic hit.
The state copy method copies all the LSP state information to a random access memory (RAM) that is not affected by the restart. After restart of a card, the LSP state on the card is recreated by copying it back from RAM. The drawback with this approach is that it requires up to 2000 bytes for each LSP and, as the number of LSPs grows, scalability in terms of RAM requirements becomes an issue. In addition, each time the RSVP state blocks are altered, there is a need to alter the graceful restart mechanism to ensure that the new fields in the state blocks are copied to RAM as well.
The protocol method with network management system (NMS) assistance requires NMS intervention to facilitate the graceful restart. Here, the NMS identifies all the LSPs passing through the card that is to be restarted. From the RSVP management information database, the NMS is able to identify the head node for each of these LSPs. After restart, the NMS contacts the ingress label edge router (LER) for each of the previously identified LSPs and initiates a modify operation on the LSPs. This causes the state to be recreated at the restarted card. The drawback of this method is that it requires NMS involvement and it impacts a large number of nodes in the network. The method also requires control plane to data plane binding and refresh hold-off operations.
The protocol methods with minimal or zero traffic hit recreates the RSVP protocol state in the restarted card by taking advantage of the RSVP refresh mechanism and adding some RSVP extensions. The protocol method with minimal traffic hit uses the mechanisms inherent in RSVP but results in over-written labels on one or more nodes. On the other hand, while the protocol method with zero traffic hit also relies on RSVP mechanisms, it will not over-write any labels and consequently should result in no traffic disruption in addition to re-establishing RSVP state on the restarted card. This approach binds the RSVP control plane to the existing data plane entries. The protocol methods with minimal or zero traffic implementation require that the router node has means to inform its neighbouring nodes to stop their refresh timeout mechanism during restart and means to determine when a link has gone down. The neighbouring nodes require means to send PATH messages to the restarting node on detection of restart completion. The restarted node also requires means to recreate the RSVP state at the restarting LSR, and means to bind the control plane RSVP state with the data plane LSP table entries.
FIG. 2 of the prior art illustrates the forwarding tables for LSPs on the nodes in the network of FIG. 1. The forwarding table on the data plane is used for switching bidirectional traffic. The forwarding table on the control plane is used for controlling the setup of connections and the direction of connection-oriented packets through the node. FIG. 2 shows a logical view of the forwarding tables 210, 220, 250, and 260 for the LSPs along with the state blocks that manage them. These tables are an over-simplification intended merely as an aid to discussing the graceful restart implementation. In FIG. 2, upRsb table 210 is Table (i) for the forward traffic incoming label embedded in RESV message; downRsb table 220 is Table (ii) for the forward traffic outgoing label embedded in RESV message; downPsb table 250 is Table (iii) for the reverse traffic incoming label embedded in PATH message; and upPsb table 260 is Table (iv) for the reverse traffic outgoing label embedded in PATH message. The upPsb table 260 (Table (iv) in FIG. 2), downPsb table 250 (Table (iii) in FIG. 2), downRsb table 220 (Table (ii) in FIG. 2), and then upRsb table 210 (Table (i) in FIG. 2) are downloaded in that order for a regular LSP setup. The forward traffic incoming label table (upRsb table) 210 contains the forward traffic inLabel entry 212 (e.g., ft.inLabel.x, ft.inLabel.y, etc.), forward traffic out interface entry 214 (e.g., ft.outlnterface.x, ft.outlnterface.y, etc.), and forward traffic pointer entry 216 (e.g., ft.Pointer.x, ft.Pointer.y, etc.). The forward traffic outgoing label table (downRsb table) 220 contains the forward traffic outLabel entry 225 (e.g., ft.outLabel.x, ft.outLabel.y, etc.). The forward traffic pointer entry points to the forward traffic out interface entry in the upRsb table; the forward traffic inLabel entry in the upRsb table; and the forward traffic outLabel entry in the downRsb table. The reverse traffic incoming label table (downPsb table) 250 contains the reverse traffic inLabel entry 252 (e.g., rt.inLabel.x, rt.inLabel.y, etc.), reverse traffic out interface entry 254 (e.g., rt.outInterface.x, rft.outInterface.y, etc.), and reverse traffic pointer entry 256 (e.g., rt.Pointer.x, rt.Pointer.y, etc.). The reverse traffic outgoing label table (upPsb table) 260 contains the reverse traffic outLabel 265 (e.g., rt.outLabel.x, rt.outLabel.y, etc.). The reverse traffic pointer entry points to the reverse traffic out interface entry in the downPsb table; the reverse traffic inLabel entry in the downPsb table; and the reverse traffic outLabel entry in the upPsb table. These tables are searched by the node processor (NP) (not shown) for matching the labels received in the PATH and RESV messages to the entries of the tables for binding the control plane to the data plane. In today's routing node, the tables are stored on both the data plane and the control plane as discussed further in the following routing node architecture.
FIG. 3 shows a prior art node 300 having a control plane 310 and a plurality of ingress card 325 and egress card 345 data planes 320 and 340. The control plane 310 having a MPLS control plane 315. The forwarding table 3150 on the MPLS control plane 315 stores the LSP states' tables, (that is, upRsb table 3151, downRsb table 3152, upPsb table 3153, and downPsb table 3154, as discussed in FIG. 2 above). The ingress card data plane 320 stores the forwarding table 3250 for said LSP states' tables, (that is, upRsb table 3251, downRsb table 3252, upPsb table 3253, and downPsb table 3254, as discussed in FIG. 2 above). The egress card data plane 340 stores the forwarding table 3450 for said LSP states' tables, (that is, upRsb table 3451, downRsb table 3452, upPsb table 3453, and downPsb table 3454, as discussed in FIG. 2 above). The MPLS control plane 315 forwarding table 3150 updates the ingress card data plane 320 forwarding table 3250 and egress card data plane 340 forwarding table 3450. In this architecture the MPLS control plane 310 is centralized for ingress and egress cards. The centralized MPLS control plane 310 and ingress and egress data planes 320 and 340 are managed separately and either data or control processor failure will not affect the entire node's operations. The ingress and egress data plane 320 and 340 uses the LSPs states' tables for data and user traffic routing in the network. The control plane 310 uses the LSPs states' tables for setting up the connections and the direction of connection-oriented packets through the network. For various reasons, such as software upgrade or control software crash, the centralized MPLS control plane 310 needs to be restarted more frequently than the data planes 320 and 340. Graceful restart at centralized MPLS control plane 310, recovers the control information on the “down” nodes without disturbing data traffic. In this architecture the forwarding table 3150 for the LSPs states are centralized for all cards and, hence, restarting the centralized MPLS control plane 310 and 315 effects the entire node's operations.
Prior art entitled “Internet draft draft-ietf-mpls-generalized-rsvp-te-09.txt, Generalized MPLS (GMPLS) signalling—RSVP-TE Extensions” by Internet Engineering Task Force (IETF) (April 2002) teaches a centralized RSVP-TE based GMPLS implementation where the LSPs states are stored on the node processor (NP) for the control plane and are centralized for all cards. The routing node architecture is as discussed in FIG. 3 above. In accordance with the prior art, for LSPs passing through a restarting node, both the upstream and downstream neighbours for all cards on the node will be affected. Upstream neighbours would not send PATH messages and disable RESV timeout while the downstream neighbours disable PATH timeout and sending of RESV refresh, (that is, both upstream and downstream neighbours are affected and detected the restart because the nodes cannot send or receive refresh packets).
The centralized RSVP-TE based GMPLS solution relies on recreation of RSVP state based on learning from their neighbours. And since all four states (upPsb, downPsb, upRsb and downRsb) were deleted, when a PATH message is received from an upstream node or a RESV message is received from a downstream node, it appears exactly the same as a new LSP creation for that node and is passed to the corresponding card on the node.
The prior art teaches a recently standardized GMPLS object that is called SUGGEST_LABEL object. When the restart capability object is sent in RSVP Hello messages to advertise a node's restart capability, then the neighbouring node sends a SUGGEST_LABEL object to the restarting node to recover its forwarding state. This is essentially the old label that the restarting node advertised before the node went down. In centralized RSVP-TE based GMPLS implementation, where all four LSPs states are stored on the node's processor (NP) for the control plane, individual card, ingress or egress, cannot be restarted.
The prior art graceful restart for centralized RSVP-TE based GMPLS implementation incorporates the Hello messages between nodes, and the restart capability object to the Hello message. This solution uses a recently standardized SUGGEST_LABEL object, at least two new timers in RSVP state machines, a new requirement to search NP (node processor) forwarding state to correlate with RSVP-TE control state, new capability to distinguish between control channel failure and genuine restart, a new provision for inter-working with fast reroute mechanism and for support of bi-directional LSP (label switched path), and other new features such as bundle, message identifier, and summary of refresh options. These new requirements for centralized RSVP-TE implementation add complexity to the graceful restart solution.
Further, the prior art introduces three new RSVP messages and objects for centralized RSVP-TE based GMPLS implementation graceful restart solution. The Hello messages are used along with bundle messages, message identifier object, and summary refresh to address RSVP scalability issues. The Hello messages are typically sent every five milliseconds to detect node failures if other such mechanisms are not available. The process consists of a node sending a Hello message and the other node responding with a Hello acknowledgement message. Changed instance values in the Hello message are used to indicate that a restart occurred. The receiver of the Hello message waits a configurable multiple of Hello intervals before assuming communication has been lost with the neighbour node. The Hello message can be included in bundle message though this is not mandatory. Another object, the restart capability object, contains the restart time and recovery time fields. The restart time is the time that the sender of the object specifies to the receiver to wait after detecting failure of RSVP communication with the sender. After this time has expired, the receiver can consider the communication severed. This value is set before any restart occurs. The recovery time value is set after the restart. The LSR or LER that has just restarted informs its neighbour that this is the amount of time it retains the forwarding state that it preserved across the restart. The restarting LSR or LER sets a timer based on recovery time value. Once recovery time expires, it deletes the LSP that doesn't have a label. The LSP states are recreated via SUGGEST_LABEL from the LSR neighbour. When recovery time value is zero, it means that the states are not preserved across the restart. When the recovery time value is set equal to “0xffffffff”, it means that the states are preserved across restart and retained till removed by means outside of the mechanisms. When the recovery time value is set to “other”, it means that no restart is detected, and LSR is operating normally. The third object is a new SUGGEST_LABEL object, which is used to inform the adjacent restarted node with the label value it provided from the sending node when the LSP was setup. It is a means of recreating the state on the restarting node.
In accordance with the prior art, after the restarted node comes up, if unable to preserve the forwarding state, it sets recovery time to zero. Otherwise it sets the recovery time to a configured value that is transmitted in the restart capability object. If the state is preserved, the restarted node sets the MPLS forwarding state holding timer to a configurable value. All RSVP states must be recreated before timer expiry. On expiry of MPLS forwarding state holding timer, the restated node searches through all forwarding plane entries, i.e., the LSPs states' tables discussed before. For each entry, the node tries to find a state in the control plane matching to RSVP. If no matching entry is found, the node deletes forwarding plane entry. When the node receives a PATH message from its neighbouring upstream node, the node searches the RSVP states in the forwarding table. If the state is found, this appears to be a refresh, and then the node treats normally. If the state is not found, and there is no SUGGEST_LABEL, the node treats as a new LSP setup, and if the state is not found and SUGGEST_LABEL is present, the node searches the forwarding tables to find an entry with matching label to the label that is suggested by the upstream node. If the entry is not found, the node treats it as a new LSP setup. If the entry is found (that is, labels are match), the node creates RSVP state and binds to forwarding plane entry. Here both incoming and outgoing labels (bi-directional) are known and fill the upstream label object with the correct label so as not to cause modification to the downstream node.
The Hello messaging between the nodes enables a node to detect that its neighbour's control plane went down. If the neighbouring node implements graceful restart, this is known from previous presence of restart capability object, then the node waits a minimum time between the restart time and local configurable timer, and then the node tries to re-establish communication with the restarted node. If the neighbour's control plane restarted, the node verifies that the neighbour preserved the state across restart via non-zero recovery time in Hello message. For each LSP where the neighbour is downstream next hop, the node inputs the original label received in label object from the neighbour into SUGGEST_LABEL object of PATH message and sends the message to the neighbour. The node holds on sending RESV messages to the neighbour until it receives the PATH message from the restarted node. If the control channel with the neighbour was lost, and the recovery time from the neighbour is non-zero, then the node treats it as communication channel restart and not as a node restart. On communication channel restart, the node sends RSVP summary refresh to the neighbour with a list of all message identifiers for all acknowledge messages.
FIG. 4 (prior art) illustrates a packet that walkthrough a portion of a network 400 for an LSP that is set up and passes through a number of LSR nodes, Node A 402, Node B 404, and Node C 406, for centralized RSVP-TE based GMPLS implementation. The node architecture is as shown in FIG. 3 where the four LSPs states' tables (upRsb, downRsb, downPsb, and upPsb tables) are centralized for all cards and hence, the restart is only performed on the node. The forward direction of traffic 405 is from Node A 402 to Node C 406, and the reverse traffic direction 495 is from node C 406 to node A 402. Node B 420 is restarted. Node A 410 and Node C 430 recognize that Node B 420 is restarted via the Hello messaging 413 and 419 between nodes, and they cancel the refresh mechanism. After a designated time, Node A 410 recognizes that Node B 420 is alive again and sends PATH message 412 to Node B 420 with the same upstream label as before but with the new SUGGEST_LABEL that is same as the label object previously sent from Node B 420 to Node A 410 before the restart. Node B 420 recreates reverse traffic outLabel entry for upPsb table and binds reverse traffic outLabel entry to upPsb table 425. To do the binding, Node B 420 searches upPsb table 425 for a label that matches the upstream lable just received. Node B 420 sends PATH message 414 to downside where reverse traffic inLabel entry for downPsb table is created. Node B searches the downPsb table 426 to find the pointer that matches the reverse traffic outLabel entry in the downPsb table that was found by searching upPsb table 425. From the entry found in the previous step, Node B 420 knows reverse traffic inLabel entry for the downPsb table for reverse direction and updates its label manager accordingly. Node B 420 then fills the PATH message 414 upstream label with this value, and sends the PATH message 414 to Node C 430. Node C 430 receives the PATH message 414 and generates RESV message 418 to Node B 420 soon thereafter. Node B 420 recreates its forward traffic outLabel entry for the downRsb table by searching the downRsb table 427 and binds the forward traffic outLabel entry to downRsb table 427. Node B 420 finds the correct entry in downRsb table 427 by searching the table for the contents of the label object sent by Node C 430. From the Node B 420 perspective, this is the outgoing label for the forward direction traffic. Node B 420 sends the RESV message 416 to the upside where forward traffic inLabel entry for the upRsb table is now created. The forward traffic inLabel entry finds its corresponding entry in upRsb table 428 by matching the SUGGEST_LABEL value received by the reverse traffic outLabel entry in upPsb table 415 from Node A 410 with the forward traffic inLabel entry in the upRsb table 428. The forward traffic inLabel entry in the upRsb table can also find its corresponding entry in the upRsb table 428 by searching the table for the forward traffic pointer entry that matches the forward traffic outLabel entry from downRsb table 427 as passed in the RESV message 418 from forward traffic outLabel entry in the downRsb table. Node B 420 now knows the forward traffic inLabel entry for upRsb table 428 and updates its label manager accordingly. Node B 420 sends the RESV message 416 to Node A 410 with its label object having the same value the SUGGEST_LABEL from Node A 410 contained that looks like regular RESV message 416 from Node A 410 perspective.
Unfortunately, the prior art providing centralized RSVP-TE based GMPLS implementation of hitless restart doesn't allow for an individual card on a node to restart, and therefore, the node restart causes loss of data/user traffic. Introducing a new object (such as, SUGGEST_LABEL object) is strongly resisted by service providers due to the inherent risk of software defects, network instability, and management complexity. Further, the SUGGEST_LABEL object is part of GMPLS (generalized multi-protocol label switching) stack and it is not suitable for use with TMPLS (traditional multi-protocol label switching). This requires customers wishing to incorporate graceful restart in their network to implement the GMPLS stack.
Prior art on protection switching in optical telecommunication network provides another solution for hitless restart, which fully protects all connections within the node at the card level. The 1+1 hitless protection switching provides one protection line card to act as a backup for one working line card, and should the working line card experience a failure, the protection line card automatically takes over and restores data flow to the network. Protection switching uses overhead bytes to identify and trigger protection switchovers. In a 1+1 hitless protection switching, each active line card has a backup (or protection) line card that can be switched into the circuit path while the primary line card is isolated in case the primary board fails. This enables individual card switchover and is accomplished by having a supervisory card that constantly monitors each card on the node and issues a switching command when necessary. Traditionally, switching has been implemented with mechanical relays. From an architecture standpoint, the relay switching solution is easy to design, but comes with inherent drawbacks. The idea is that identical signalling streams are transmitted out over two physical ports. The two receivers on the far side listen only on the working port, known as the primary port. When certain conditions are detected, such as loss of frame, loss of signal, and signal degradation, the receiver simply begins listening on the protection (or backup) port. When transmitting data, both the working port and the protection port send duplicate frames. The transmitting side makes no adjustments or configuration changes during or after protection switching failover.
Thus, the prior art on hitless protection switching for optical telecommunication networks provides graceful restart. However, it requires redundancy in hardware and software resources. These resources are implemented in a one-to-one and one-to-many backup. The 1+1 hitless protection switching is not a centralized implementation of graceful restart, but rather distributed over the line cards, which enable individual card switchover to backup line card with no impact on the entire node's operations. Therefore, for hitless protection switching, redundant hardware and software resources are required for implementing protection switchovers, which results in increased capital and operational costs.
Accordingly, there is a need for the development of improved routing node architecture and methods for hitless graceful restart for an RSVP-TE (resource reservation protocol-traffic engineering) based MPLS (multi-protocol label switching) that would overcome the shortcomings and limitations of the prior art.